Structured Prediction with Perceptron: Theory and Algorithms

Structured prediction problem is a special case of machine learning problem where both the inputs and outputs are structures such as sequences, trees, and graphs, rather than plain single labels or values. Many important natural language processing (NLP) problems are structured prediction problems, including Part-Of-Speech tagging, parsing, and machine translation. This survey investigates how perceptron, the simplest supervised machine learning algorithm, can be adapted to handle structured inputs and outputs. In particular, for the estimation problem that searches for the right output structure, introducing structures leads to combinatorial explosion and the inference becomes extremely time-consuming. Various dynamic programming techniques and approximations are developed to speed up the search. For the learning problem, especially supervised learning with perceptron, searching structures with approximations makes the standard perceptron learning algorithm unconvergable. To address this problem, new learning methods are invented for structured learning. In addition, complicated structured prediction problems usually involve unobserved structures as hidden variables. A latent variable perceptron model is also discussed to handle those problems.